264 research outputs found
Optimal Information Retrieval with Complex Utility Functions
Existing retrieval models all attempt to optimize one single utility function, which is often based on the topical relevance of a document with respect to a query. In real applications, retrieval involves more complex utility functions that may involve preferences on several different dimensions. In this paper, we present a general optimization framework for retrieval with complex utility functions. A query language is designed according to this framework to enable users to submit complex queries. We propose an efficient algorithm for retrieval with complex utility functions based on the a-priori algorithm. As a case study, we apply our algorithm to a complex utility retrieval problem in distributed IR. Experiment results show that our algorithm allows for flexible tradeoff between multiple retrieval criteria. Finally, we study the efficiency issue of our algorithm on simulated data
Adapting Sequence to Sequence models for Text Normalization in Social Media
Social media offer an abundant source of valuable raw data, however informal
writing can quickly become a bottleneck for many natural language processing
(NLP) tasks. Off-the-shelf tools are usually trained on formal text and cannot
explicitly handle noise found in short online posts. Moreover, the variety of
frequently occurring linguistic variations presents several challenges, even
for humans who might not be able to comprehend the meaning of such posts,
especially when they contain slang and abbreviations. Text Normalization aims
to transform online user-generated text to a canonical form. Current text
normalization systems rely on string or phonetic similarity and classification
models that work on a local fashion. We argue that processing contextual
information is crucial for this task and introduce a social media text
normalization hybrid word-character attention-based encoder-decoder model that
can serve as a pre-processing step for NLP applications to adapt to noisy text
in social media. Our character-based component is trained on synthetic
adversarial examples that are designed to capture errors commonly found in
online user-generated text. Experiments show that our model surpasses neural
architectures designed for text normalization and achieves comparable
performance with state-of-the-art related work.Comment: Accepted at the 13th International AAAI Conference on Web and Social
Media (ICWSM 2019
To Asymmetry and Beyond: Structured Pruning of Sequence to Sequence Models for Improved Inference Efficiency
Sequence-to-sequence language models can be used to produce abstractive
summaries which are coherent, relevant, and concise. Still, model sizes can
make deployment in latency-sensitive or web-scale implementations difficult.
This paper studies the relationship between model size, structured pruning,
inference efficiency, and summarization accuracy on widely used summarization
datasets. We show that model accuracy is tied to the encoder size while
inference efficiency is connected to the decoder. Using asymmetric pruning can
lead to nearly 3x improvement in inference latency with ~1 point loss in
Rouge-2. Moreover, we find both the average degradation and the role of
asymmetry to be consistent across model sizes and variations in datasets.Comment: SustaiNLP2023 @ ACL 2023,9 pages, 6 figures, 33 table
Comprehensive Review of Opinion Summarization
The abundance of opinions on the web has kindled the study of opinion summarization over the last few years. People have introduced various techniques and paradigms to solving this special task. This survey attempts to systematically investigate the different techniques and approaches used in opinion summarization. We provide a multi-perspective classification of the approaches used and highlight some of the key weaknesses of these approaches. This survey also covers evaluation techniques and data sets used in studying the opinion summarization problem. Finally, we provide insights into some of the challenges that are left to be addressed as this will help set the trend for future research in this area.unpublishednot peer reviewe
Noise-Robust Dense Retrieval via Contrastive Alignment Post Training
The success of contextual word representations and advances in neural
information retrieval have made dense vector-based retrieval a standard
approach for passage and document ranking. While effective and efficient,
dual-encoders are brittle to variations in query distributions and noisy
queries. Data augmentation can make models more robust but introduces overhead
to training set generation and requires retraining and index regeneration. We
present Contrastive Alignment POst Training (CAPOT), a highly efficient
finetuning method that improves model robustness without requiring index
regeneration, the training set optimization, or alteration. CAPOT enables
robust retrieval by freezing the document encoder while the query encoder
learns to align noisy queries with their unaltered root. We evaluate CAPOT
noisy variants of MSMARCO, Natural Questions, and Trivia QA passage retrieval,
finding CAPOT has a similar impact as data augmentation with none of its
overhead.Comment: 8 pages, 6 figures, 30 table
Competence-Based Analysis of Language Models
Despite the recent success of large pretrained language models (LMs) on a
variety of prompting tasks, these models can be alarmingly brittle to small
changes in inputs or application contexts. To better understand such behavior
and motivate the design of more robust LMs, we propose a general experimental
framework, CALM (Competence-based Analysis of Language Models), where targeted
causal interventions are utilized to damage an LM's internal representation of
various linguistic properties in order to evaluate its use of each
representation in performing a given task. We implement these interventions as
gradient-based adversarial attacks, which (in contrast to prior causal probing
methodologies) are able to target arbitrarily-encoded representations of
relational properties, and carry out a case study of this approach to analyze
how BERT-like LMs use representations of several relational properties in
performing associated relation prompting tasks. We find that, while the
representations LMs leverage in performing each task are highly entangled, they
may be meaningfully interpreted in terms of the tasks where they are most
utilized; and more broadly, that CALM enables an expanded scope of inquiry in
LM analysis that may be useful in predicting and explaining weaknesses of
existing LMs
Quick Dense Retrievers Consume KALE: Post Training Kullback Leibler Alignment of Embeddings for Asymmetrical dual encoders
In this paper, we consider the problem of improving the inference latency of
language model-based dense retrieval systems by introducing structural
compression and model size asymmetry between the context and query encoders.
First, we investigate the impact of pre and post-training compression on the
MSMARCO, Natural Questions, TriviaQA, SQUAD, and SCIFACT, finding that
asymmetry in the dual encoders in dense retrieval can lead to improved
inference efficiency. Knowing this, we introduce Kullback Leibler Alignment of
Embeddings (KALE), an efficient and accurate method for increasing the
inference efficiency of dense retrieval methods by pruning and aligning the
query encoder after training. Specifically, KALE extends traditional Knowledge
Distillation after bi-encoder training, allowing for effective query encoder
compression without full retraining or index generation. Using KALE and
asymmetric training, we can generate models which exceed the performance of
DistilBERT despite having 3x faster inference.Comment: 8 pages, 4 figures, 30 table
- …